Maybe I should add that this unit is used for backup. Backing up about 50 clients right now.
I am telling you this because of this line on vmwares info page about the error seen in the vobd.log:
Also check if these messages are generated when there were any scheduled tasks, such as backups, replications, etc., as these can also cause intermittent performance hits.
But I tried to turn the backup off, the entries in the /var/log/vmkwarning.log did continue (the aprox 10 min cycle), so I don´t have to much trust in this being just the backup running :/ But I wanted to tell you everything I have been testing my self!
The vobd.log as well
2015-01-20T00:00:43.950Z: [scsiCorrelator] 1765367276456us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T2:L0 performance has improved. I/O latency reduced from 5301 microseconds to 4421 microseconds.
2015-01-20T00:00:43.950Z: [scsiCorrelator] 1765367277003us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T2:L0 performance has improved. I/O latency reduced from 5301 microseconds to 4421 microseconds.
2015-01-20T00:40:51.159Z: [scsiCorrelator] 1767774485074us: [vob.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 12197 microseconds to 393127 microseconds.
2015-01-20T00:40:51.159Z: [scsiCorrelator] 1767774485640us: [esx.problem.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 12197 microseconds to 393127 microseconds.
2015-01-20T00:46:12.361Z: [scsiCorrelator] 1768095687062us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 393127 microseconds to 77498 microseconds.
2015-01-20T00:46:12.361Z: [scsiCorrelator] 1768095687609us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 393127 microseconds to 77498 microseconds.
2015-01-20T01:26:44.097Z: [scsiCorrelator] 1770527423001us: [vob.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 12199 microseconds to 368277 microseconds.
2015-01-20T01:26:44.097Z: [scsiCorrelator] 1770527423596us: [esx.problem.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 12199 microseconds to 368277 microseconds.
2015-01-20T01:30:06.129Z: [scsiCorrelator] 1770729454694us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 368277 microseconds to 73338 microseconds.
2015-01-20T01:30:06.129Z: [scsiCorrelator] 1770729455288us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 368277 microseconds to 73338 microseconds.
2015-01-20T01:51:28.673Z: [scsiCorrelator] 1772011998774us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 73338 microseconds to 23880 microseconds.
2015-01-20T01:51:28.673Z: [scsiCorrelator] 1772011999383us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 73338 microseconds to 23880 microseconds.
2015-01-20T03:25:02.215Z: [scsiCorrelator] 1777625540512us: [vob.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 12216 microseconds to 371844 microseconds.
2015-01-20T03:25:02.215Z: [scsiCorrelator] 1777625541103us: [esx.problem.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 12216 microseconds to 371844 microseconds.
2015-01-20T03:26:46.280Z: [scsiCorrelator] 1777729605765us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 371844 microseconds to 73316 microseconds.
2015-01-20T03:26:46.280Z: [scsiCorrelator] 1777729606326us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 371844 microseconds to 73316 microseconds.
2015-01-20T03:58:40.271Z: [scsiCorrelator] 1779643597414us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 73316 microseconds to 24096 microseconds.
2015-01-20T03:58:40.272Z: [scsiCorrelator] 1779643598052us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 73316 microseconds to 24096 microseconds.
2015-01-21T02:43:49.038Z: [scsiCorrelator] 1861552364053us: [vob.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11838 microseconds to 357325 microseconds.
2015-01-21T02:43:49.038Z: [scsiCorrelator] 1861552364694us: [esx.problem.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11838 microseconds to 357325 microseconds.
2015-01-21T02:44:38.848Z: [scsiCorrelator] 1861602173474us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 357325 microseconds to 71302 microseconds.
2015-01-21T02:44:38.848Z: [scsiCorrelator] 1861602174108us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 357325 microseconds to 71302 microseconds.
2015-01-21T03:03:03.195Z: [scsiCorrelator] 1862706521047us: [vob.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11839 microseconds to 389600 microseconds.
2015-01-21T03:03:03.195Z: [scsiCorrelator] 1862706521624us: [esx.problem.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11839 microseconds to 389600 microseconds.
2015-01-21T03:04:49.393Z: [scsiCorrelator] 1862812718887us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 389600 microseconds to 76579 microseconds.
2015-01-21T03:04:49.393Z: [scsiCorrelator] 1862812719508us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 389600 microseconds to 76579 microseconds.
2015-01-21T03:17:44.791Z: [scsiCorrelator] 1863588117001us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 76579 microseconds to 23479 microseconds.
2015-01-21T03:17:44.791Z: [scsiCorrelator] 1863588117577us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 76579 microseconds to 23479 microseconds.
2015-01-21T03:57:11.318Z: [scsiCorrelator] 1865954643900us: [vob.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11851 microseconds to 368958 microseconds.
2015-01-21T03:57:11.318Z: [scsiCorrelator] 1865954644492us: [esx.problem.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11851 microseconds to 368958 microseconds.
2015-01-21T03:58:47.330Z: [scsiCorrelator] 1866050655796us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 368958 microseconds to 72969 microseconds.
2015-01-21T03:58:47.330Z: [scsiCorrelator] 1866050656403us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 368958 microseconds to 72969 microseconds.
2015-01-21T04:03:36.648Z: [scsiCorrelator] 1866339974393us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 72969 microseconds to 23507 microseconds.
2015-01-21T04:03:36.648Z: [scsiCorrelator] 1866339974994us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 72969 microseconds to 23507 microseconds.
2015-01-21T18:59:14.845Z: [scsiCorrelator] 1920078170629us: [vob.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11510 microseconds to 379288 microseconds.
2015-01-21T18:59:14.845Z: [scsiCorrelator] 1920078171219us: [esx.problem.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11510 microseconds to 379288 microseconds.
2015-01-21T19:01:16.836Z: [scsiCorrelator] 1920200162422us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 379288 microseconds to 75549 microseconds.
2015-01-21T19:01:16.837Z: [scsiCorrelator] 1920200163039us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 379288 microseconds to 75549 microseconds.
2015-01-21T19:07:19.007Z: [scsiCorrelator] 1920562333444us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 75549 microseconds to 22669 microseconds.
2015-01-21T19:07:19.008Z: [scsiCorrelator] 1920562334057us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 75549 microseconds to 22669 microseconds.
2015-01-21T20:31:34.433Z: [scsiCorrelator] 1925617758608us: [vob.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11519 microseconds to 365880 microseconds.
2015-01-21T20:31:34.433Z: [scsiCorrelator] 1925617759236us: [esx.problem.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11519 microseconds to 365880 microseconds.
2015-01-21T20:32:09.254Z: [scsiCorrelator] 1925652579533us: [vob.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11519 microseconds to 745015 microseconds.
2015-01-21T20:32:09.254Z: [scsiCorrelator] 1925652580154us: [esx.problem.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11519 microseconds to 745015 microseconds.
2015-01-21T20:32:32.628Z: [scsiCorrelator] 1925675954357us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 745015 microseconds to 146457 microseconds.
2015-01-21T20:32:32.628Z: [scsiCorrelator] 1925675954939us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 745015 microseconds to 146457 microseconds.
2015-01-21T20:32:44.389Z: [scsiCorrelator] 1925687714790us: [vob.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11520 microseconds to 358113 microseconds.
2015-01-21T20:32:44.389Z: [scsiCorrelator] 1925687715354us: [esx.problem.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11520 microseconds to 358113 microseconds.
2015-01-21T20:33:10.953Z: [scsiCorrelator] 1925714279444us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 358113 microseconds to 69812 microseconds.
2015-01-21T20:33:10.954Z: [scsiCorrelator] 1925714280043us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 358113 microseconds to 69812 microseconds.
2015-01-21T20:35:31.577Z: [scsiCorrelator] 1925854902790us: [vob.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11520 microseconds to 373986 microseconds.
2015-01-21T20:35:31.577Z: [scsiCorrelator] 1925854903386us: [esx.problem.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11520 microseconds to 373986 microseconds.
2015-01-21T20:37:05.872Z: [scsiCorrelator] 1925949198202us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 373986 microseconds to 74350 microseconds.
2015-01-21T20:37:05.872Z: [scsiCorrelator] 1925949198790us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 373986 microseconds to 74350 microseconds.
2015-01-21T20:45:22.808Z: [scsiCorrelator] 1926446134219us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 74350 microseconds to 22416 microseconds.
2015-01-21T20:45:22.808Z: [scsiCorrelator] 1926446134824us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 74350 microseconds to 22416 microseconds.
2015-01-21T22:26:11.311Z: [scsiCorrelator] 1932494637316us: [vob.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11526 microseconds to 360319 microseconds.
2015-01-21T22:26:11.311Z: [scsiCorrelator] 1932494637929us: [esx.problem.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11526 microseconds to 360319 microseconds.
2015-01-21T22:34:26.519Z: [scsiCorrelator] 1932989844615us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 360319 microseconds to 71603 microseconds.
2015-01-21T22:34:26.519Z: [scsiCorrelator] 1932989845238us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 360319 microseconds to 71603 microseconds.
2015-01-21T22:43:04.185Z: [scsiCorrelator] 1933507511075us: [vob.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11526 microseconds to 357249 microseconds.
2015-01-21T22:43:04.185Z: [scsiCorrelator] 1933507511706us: [esx.problem.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11526 microseconds to 357249 microseconds.
2015-01-21T22:49:38.692Z: [scsiCorrelator] 1933902018022us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 357249 microseconds to 70490 microseconds.
2015-01-21T22:49:38.692Z: [scsiCorrelator] 1933902018628us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 357249 microseconds to 70490 microseconds.
2015-01-21T23:26:05.258Z: [scsiCorrelator] 1936088583913us: [vob.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11528 microseconds to 355514 microseconds.
2015-01-21T23:26:05.258Z: [scsiCorrelator] 1936088584498us: [esx.problem.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11528 microseconds to 355514 microseconds.
2015-01-21T23:30:13.597Z: [scsiCorrelator] 1936336922530us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 355514 microseconds to 70830 microseconds.
2015-01-21T23:30:13.597Z: [scsiCorrelator] 1936336923129us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 355514 microseconds to 70830 microseconds.
2015-01-22T00:10:44.107Z: [scsiCorrelator] 1938767433400us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 70830 microseconds to 22576 microseconds.
2015-01-22T00:10:44.108Z: [scsiCorrelator] 1938767434023us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 70830 microseconds to 22576 microseconds.
2015-01-22T03:17:28.580Z: [scsiCorrelator] 1949971906415us: [vob.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11530 microseconds to 348863 microseconds.
2015-01-22T03:17:28.581Z: [scsiCorrelator] 1949971907047us: [esx.problem.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11530 microseconds to 348863 microseconds.
2015-01-22T03:23:52.833Z: [scsiCorrelator] 1950356158615us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 348863 microseconds to 68635 microseconds.
2015-01-22T03:23:52.833Z: [scsiCorrelator] 1950356159211us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 348863 microseconds to 68635 microseconds.
2015-01-22T03:23:54.282Z: [scsiCorrelator] 1950357608044us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 68635 microseconds to 22995 microseconds.
2015-01-22T03:23:54.282Z: [scsiCorrelator] 1950357608675us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 68635 microseconds to 22995 microseconds.
2015-01-22T03:48:47.486Z: [scsiCorrelator] 1951850812258us: [vob.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11530 microseconds to 376890 microseconds.
2015-01-22T03:48:47.486Z: [scsiCorrelator] 1951850812929us: [esx.problem.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11530 microseconds to 376890 microseconds.
2015-01-22T04:00:17.871Z: [scsiCorrelator] 1952541197205us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 376890 microseconds to 74290 microseconds.
2015-01-22T04:00:17.871Z: [scsiCorrelator] 1952541197837us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 376890 microseconds to 74290 microseconds.
2015-01-22T04:38:01.784Z: [scsiCorrelator] 1954805109594us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 74290 microseconds to 22432 microseconds.
2015-01-22T04:38:01.784Z: [scsiCorrelator] 1954805110193us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 74290 microseconds to 22432 microseconds.
2015-01-22T05:26:05.282Z: [scsiCorrelator] 1957688608008us: [vob.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11536 microseconds to 348785 microseconds.
2015-01-22T05:26:05.282Z: [scsiCorrelator] 1957688608587us: [esx.problem.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T1:L0 performance has deteriorated. I/O latency increased from average value of 11536 microseconds to 348785 microseconds.
2015-01-22T05:28:03.138Z: [scsiCorrelator] 1957806463877us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 348785 microseconds to 68859 microseconds.
2015-01-22T05:28:03.138Z: [scsiCorrelator] 1957806464437us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 348785 microseconds to 68859 microseconds.
2015-01-22T06:24:29.841Z: [scsiCorrelator] 1961193166618us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 68859 microseconds to 22911 microseconds.
2015-01-22T06:24:29.841Z: [scsiCorrelator] 1961193167218us: [esx.clear.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T1:L0 performance has improved. I/O latency reduced from 68859 microseconds to 22911 microseconds.