|
|
Objective: We want to predict whether the hard drive is going to fail or not based on various indicators of drive reliability. We also want to detect categorical outliers in the data.
|
|
|
|
|
|
1. License: Free to use with following constraints.
|
|
|
A. We are encouraged to Cite Backblaze as the source (not a mandatory requirement).
|
|
|
B. We accept that we are solely responsible for how we use the data.
|
|
|
C. We do not sell this data to anyone, it is free.
|
|
|
|
|
|
2. Data Source: https://www.backblaze.com/hard-drive-test-data.html
|
|
|
|
|
|
3. Field Meaning:
|
|
|
A. SMART_1_Raw: Read Error Rate -> Stores data related to the rate of hardware read errors that occurred when reading data from a disk surface.
|
|
|
B. SMART_2_Raw: Reallocated Sectors Count —> Typically, raw value normally represents a count of the bad sectors that have been found and remapped. Thus, the higher the attribute value, the more sectors the drive has had to reallocate.
|
|
|
C. SMART_3_Raw: Power-On Hours —> Raw value of this means total count of hours the drive is on power-on state.
|
|
|
D. SMART_4_Raw: Temperature Celsius —> Current internal temperature.
|
|
|
E. SMART_5_Raw: Current Pending Sector Count —> Shows the count of "unstable" sectors.
|
|
|
|
|
|
F. DiskFailure: ‘Yes’ means failure, ‘No’ means condition of the hard drive is fine.
|
|
|
G. Date: Refers to snapshot of hard drive in that particular day.
|
|
|
H. Model: Manufacturer-assigned model number of the hard drive.
|
|
|
I. SerialNumber: Manufacturer-assigned serial number of the hard drive.
|
|
|
J. CapacityBytes: Refers to hard drive capacity in bytes.
|
|
|
|
|
|
4. Parameter Selection:
|
|
|
A. Dashboard Usage: Predict Categorical Fields
|
|
|
Settings:
|
|
|
1) Search Command: | inputlookup disk_failures.csv | eventstats max(SMART_1_Raw) as max1 min(SMART_1_Raw) as min1 | eventstats max(SMART_2_Raw) as max2 min(SMART_2_Raw) as min2 | eventstats max(SMART_3_Raw) as max3 min(SMART_3_Raw) as min3 | eventstats max(SMART_4_Raw) as max4 min(SMART_4_Raw) as min4 | eventstats max(SMART_5_Raw) as max5 min(SMART_5_Raw) as min5 | eval SMART_1_Transformed = (SMART_1_Raw - min1)/(max1-min1) | eval SMART_2_Transformed = (SMART_2_Raw - min2)/(max2-min2) | eval SMART_3_Transformed = (SMART_3_Raw - min3)/(max3-min3) | eval SMART_4_Transformed = (SMART_4_Raw - min4)/(max4-min4) | eval SMART_5_Transformed = (SMART_5_Raw - min5)/(max5-min5) | table Date Model CapacityBytes SerialNumber DiskFailure SMART_1_Raw SMART_1_Transformed SMART_2_Raw SMART_2_Transformed SMART_3_Raw SMART_3_Transformed SMART_4_Raw SMART_4_Transformed SMART_5_Raw SMART_5_Transformed
|
|
|
2) Field to predict: DiskFailure
|
|
|
3) Fields to use for predicting: Model, SMART_1_Transformed, SMART_2_Transformed, SMART_3_Transformed, SMART_4_Transformed, SMART_5_Transformed
|
|
|
|
|
|
B. Dashboard: Detect Categorical Outliers
|
|
|
Settings:
|
|
|
1) Fields to analyze: Model, CapacityBytes, DiskFailure, SerialNumber
|
|
|
|
|
|
C. Dashboard: Cluster Numeric Fields
|
|
|
Settings:
|
|
|
0) Search: | inputlookup disk_failures.csv | eventstats max(SMART_1_Raw) as max1 min(SMART_1_Raw) as min1 | eventstats max(SMART_2_Raw) as max2 min(SMART_2_Raw) as min2 | eventstats max(SMART_3_Raw) as max3 min(SMART_3_Raw) as min3 | eventstats max(SMART_4_Raw) as max4 min(SMART_4_Raw) as min4 | eventstats max(SMART_5_Raw) as max5 min(SMART_5_Raw) as min5 | eval SMART_1_Transformed = (SMART_1_Raw - min1)/(max1-min1) | eval SMART_2_Transformed = (SMART_2_Raw - min2)/(max2-min2) | eval SMART_3_Transformed = (SMART_3_Raw - min3)/(max3-min3) | eval SMART_4_Transformed = (SMART_4_Raw - min4)/(max4-min4) | eval SMART_5_Transformed = (SMART_5_Raw - min5)/(max5-min5) | table Date Model CapacityBytes SerialNumber DiskFailure SMART_1_Raw SMART_1_Transformed SMART_2_Raw SMART_2_Transformed SMART_3_Raw SMART_3_Transformed SMART_4_Raw SMART_4_Transformed SMART_5_Raw SMART_5_Transformed
|
|
|
1) Model name: disk_failures
|
|
|
2) Fields to preprocess: SMART_1_Raw, SMART_1_Transformed, SMART_2_Raw, SMART_2_Transformed, SMART_3_Raw, SMART_3_Transformed, SMART_4_Raw, SMART_4_Transformed, SMART_5_Raw, SMART_5_Transformed
|
|
|
3) Apply StandardScaler
|
|
|
4) Apply PCA to reduce to 4 fields
|
|
|
5) Algorithm: Birch
|
|
|
6) Fields to use: PC_1, PC_2, PC_3, PC_4
|
|
|
7) K: 3
|
