[Hardware] Работа с iSCSI HBA ServerEngines 10Gbe (be3) в HP ProLiant BL460c G7
Alex Moskalenko
mav на elserv.msk.su
Сб Апр 7 21:13:12 MSK 2012
Здравствуйте!
Подскажите пожалуйста, получалось ли у кого-нибудь работать с
iSCSI-инициатором из HP Proliant BL460c G7 на наших дистрибутивах?
Интересует работа именно через be2iscsi, так как через стандартный
iscsi-tcp все работает.
На данный момент у меня на всех испробованных ядрах
(kernel-image-ovz-el-2.6.32-alt57,
kernel-image-std-def-3.0.26-alt0.M60P.1,
kernel-image-std-def-3.2.13-alt1) получается одна и та же картина:
Устройство опознается, загружается модуль be2iscsi;
iscsiadm -m iface устройства be2iscsi видит;
Попытка подключиться к таргету (как IET, так и LIO) заканчивается
отказом сетевого интерфейса (ethX) этого адаптера и подвисанием
iscsi-интерфейса (процессы /lib/udev/scsi_id висят в uninterruptible
sleep). Соответственно, доступ к серверу по сети пропадает.
То же самое более подробно:
root на blade2 log]# uname -a
Linux blade2.office 3.2.13-std-def-alt1 #1 SMP Tue Mar 27 09:33:17 UTC
2012 x86_64 GNU/Linux
[root на blade2 log]# lspci -v
02:00.0 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC
(be3) (rev 01)
Subsystem: Hewlett-Packard Company NC553i 10Gb 2-port FlexFabric
Converged Network Adapter
Flags: bus master, fast devsel, latency 0, IRQ 24
Memory at fbff0000 (64-bit, non-prefetchable) [size=16K]
Memory at fbfc0000 (64-bit, non-prefetchable) [size=128K]
Memory at fbfa0000 (64-bit, non-prefetchable) [size=128K]
[virtual] Expansion ROM at f8000000 [disabled] [size=256K]
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI-X: Enable+ Count=32 Masked-
Capabilities: [c0] Express Endpoint, MSI 00
Capabilities: [b8] Vital Product Data
Capabilities: [100] Advanced Error Reporting
Capabilities: [180] Single Root I/O Virtualization (SR-IOV)
Capabilities: [160] Alternative Routing-ID Interpretation (ARI)
Capabilities: [168] Device Serial Number 44-1e-a1-ff-fe-52-11-00
Capabilities: [12c] #17
Kernel driver in use: be2net
Kernel modules: be2net
02:00.1 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC
(be3) (rev 01)
Subsystem: Hewlett-Packard Company NC553i 10Gb 2-port FlexFabric
Converged Network Adapter
Flags: bus master, fast devsel, latency 0, IRQ 34
Memory at fbf90000 (64-bit, non-prefetchable) [size=16K]
Memory at fbf60000 (64-bit, non-prefetchable) [size=128K]
Memory at fbf40000 (64-bit, non-prefetchable) [size=128K]
[virtual] Expansion ROM at f8040000 [disabled] [size=256K]
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI-X: Enable+ Count=32 Masked-
Capabilities: [c0] Express Endpoint, MSI 00
Capabilities: [b8] Vital Product Data
Capabilities: [100] Advanced Error Reporting
Capabilities: [180] Single Root I/O Virtualization (SR-IOV)
Capabilities: [160] Alternative Routing-ID Interpretation (ARI)
Capabilities: [168] Device Serial Number 44-1e-a1-ff-fe-52-11-00
Capabilities: [12c] #17
Kernel driver in use: be2net
Kernel modules: be2net
02:00.2 Mass storage controller: Emulex Corporation OneConnect 10Gb
iSCSI Initiator (be3) (rev 01)
Subsystem: Hewlett-Packard Company Device 3315
Flags: bus master, fast devsel, latency 0, IRQ 35
Memory at fbf30000 (64-bit, non-prefetchable) [size=16K]
Memory at fbf00000 (64-bit, non-prefetchable) [size=128K]
Memory at fbee0000 (64-bit, non-prefetchable) [size=128K]
[virtual] Expansion ROM at f8080000 [disabled] [size=256K]
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI-X: Enable+ Count=32 Masked-
Capabilities: [c0] Express Endpoint, MSI 00
Capabilities: [b8] Vital Product Data
Capabilities: [100] Advanced Error Reporting
Capabilities: [180] Single Root I/O Virtualization (SR-IOV)
Capabilities: [160] Alternative Routing-ID Interpretation (ARI)
Capabilities: [168] Device Serial Number 44-1e-a1-ff-fe-52-11-00
Capabilities: [12c] #17
Kernel driver in use: be2iscsi
Kernel modules: be2iscsi
02:00.3 Mass storage controller: Emulex Corporation OneConnect 10Gb
iSCSI Initiator (be3) (rev 01)
Subsystem: Hewlett-Packard Company Device 3315
Flags: bus master, fast devsel, latency 0, IRQ 36
Memory at fbed0000 (64-bit, non-prefetchable) [size=16K]
Memory at fbea0000 (64-bit, non-prefetchable) [size=128K]
Memory at fbe80000 (64-bit, non-prefetchable) [size=128K]
[virtual] Expansion ROM at f80c0000 [disabled] [size=256K]
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI-X: Enable+ Count=32 Masked-
Capabilities: [c0] Express Endpoint, MSI 00
Capabilities: [b8] Vital Product Data
Capabilities: [100] Advanced Error Reporting
Capabilities: [180] Single Root I/O Virtualization (SR-IOV)
Capabilities: [160] Alternative Routing-ID Interpretation (ARI)
Capabilities: [168] Device Serial Number 44-1e-a1-ff-fe-52-11-00
Capabilities: [12c] #17
Kernel driver in use: be2iscsi
Kernel modules: be2iscsi
[root на blade2 ~]# iscsiadm --version
iscsiadm version 2.0-872
[root на blade2 ~]# iscsiadm -m iface -I be2iscsi.44:1e:a1:52:11:01
# BEGIN RECORD 2.0-872
iface.iscsi_ifacename = be2iscsi.44:1e:a1:52:11:01
iface.net_ifacename = <empty>
iface.ipaddress = 10.10.10.130
iface.hwaddress = 44:1e:a1:52:11:01
iface.transport_name = be2iscsi
iface.initiatorname = iqn.2012-04.local.iscsi:blade2
# END RECORD
iscsiadm -m node -I be2iscsi.44:1e:a1:52:11:01 --targetname
iqn.2012-04.local.iscsi.test:test -p 10.10.10.1 --login
После этой команды сеть обычно отваливается. С консоли видно, что
команда с задержкой секунд 20 отрабатывает. В системе появляются диски
соответственно LUNам на таргете. Любая попытка обращения к ним приводит
к отправке процесса в uninterruptible sleep.
В messages после попытки подключения таргета появляются следующие сообщения:
Apr 7 18:58:02 blade2 kernel: [ 136.525847]
(beiscsi_process_cq():1896):CQ Error notification for cmd.. code 18 cid 0x0
Apr 7 18:58:03 blade2 iscsid: Connection1:0 to [target:
iqn.2012-04.local.iscsi.test:test, portal: 10.10.10.1,3260] through [ifac
e: be2iscsi.44:1e:a1:52:11:01] is operational now
Apr 7 18:58:23 blade2 kernel: [ 157.379351]
(beiscsi_process_cq():1896):CQ Error notification for cmd.. code 18 cid 0x0
Apr 7 18:58:44 blade2 kernel: [ 178.402364]
(beiscsi_process_cq():1896):CQ Error notification for cmd.. code 18 cid 0x0
Apr 7 18:58:59 blade2 kernel: [ 193.411235] connection1:0: detected
conn error (1021)
Apr 7 18:59:00 blade2 iscsid: Kernel reported iSCSI connection 1:0
error (1021) state (3)
Apr 7 18:59:02 blade2 kernel: [ 196.475407] scsi 1:0:0:0:
Direct-Access LIO-ORG FILEIO 4.0 PQ: 0 ANSI: 5
Apr 7 18:59:02 blade2 kernel: [ 196.475663] sd 1:0:0:0: Attached scsi
generic sg1 type 0
Apr 7 18:59:02 blade2 kernel: [ 196.476116] sd 1:0:0:0: [sdb] 2097153
512-byte logical blocks: (1.07 GB/1.00 GiB)
Apr 7 18:59:02 blade2 kernel: [ 196.476460] scsi 1:0:0:1:
Direct-Access LIO-ORG FILEIO 4.0 PQ: 0 ANSI: 5
Apr 7 18:59:02 blade2 kernel: [ 196.476688] sd 1:0:0:1: Attached scsi
generic sg2 type 0
Apr 7 18:59:02 blade2 kernel: [ 196.477255] sd 1:0:0:1: [sdc] 2097153
512-byte logical blocks: (1.07 GB/1.00 GiB)
Apr 7 18:59:02 blade2 kernel: [ 196.477672] sd 1:0:0:0: [sdb] Write
Protect is off
Apr 7 18:59:02 blade2 kernel: [ 196.478294] sd 1:0:0:0: [sdb] Write
cache: disabled, read cache: enabled, doesn't support DPO or FUA
Apr 7 18:59:02 blade2 kernel: [ 196.478554] sd 1:0:0:1: [sdc] Write
Protect is off
Apr 7 18:59:02 blade2 kernel: [ 196.479093] sd 1:0:0:1: [sdc] Write
cache: disabled, read cache: enabled, doesn't support DPO or FUA
Apr 7 18:59:02 blade2 kernel: [ 196.481398] sdb: sdb1
Apr 7 18:59:02 blade2 kernel: [ 196.482642] sdc: sdc1
Apr 7 18:59:02 blade2 kernel: [ 196.484413] sd 1:0:0:0: [sdb] Attached
SCSI disk
Apr 7 18:59:02 blade2 kernel: [ 196.484797] sd 1:0:0:1: [sdc] Attached
SCSI disk
Apr 7 18:59:03 blade2 iscsid: connection1:0 is operational after
recovery (1 attempts)
Apr 7 18:59:03 blade2 kernel: [ 197.347283] be2net 0000:02:00.1: UE
Detected!!
Apr 7 18:59:04 blade2 kernel: [ 197.347289] be2net 0000:02:00.1: UE:
RXULP2 bit set
Apr 7 18:59:04 blade2 kernel: [ 197.347293] be2net 0000:02:00.1: Error
in Card Detected! Cannot issue commands
Apr 7 18:59:04 blade2 kernel: [ 197.451236] be2net 0000:02:00.0: UE
Detected!!
Apr 7 18:59:04 blade2 kernel: [ 197.451240] be2net 0000:02:00.0: UE:
RXULP2 bit set
Apr 7 18:59:04 blade2 kernel: [ 197.451244] be2net 0000:02:00.0: Error
in Card Detected! Cannot issue commands
Apr 7 18:59:12 blade2 kernel: [ 206.499211] connection1:0: ping
timeout of 5 secs expired, recv timeout 5, last rx 4294940032, last ping
4294
941282, now 4294942536
Apr 7 18:59:12 blade2 kernel: [ 206.499224] connection1:0: detected
conn error (1011)
Apr 7 18:59:13 blade2 iscsid: Kernel reported iSCSI connection 1:0
error (1011) state (3)
При появлении сообщений от be2net перестает работать сетевой интерфейс.
Ядро kernel-image-std-def-3.0.26-alt0.M60P.1 в дополнение к этому еще
пишет следующие сообщения от be2net
Apr 7 20:47:13 blade2 kernel: [ 76.027096] be2net 0000:02:00.0: Error
in cmd completion - opcode 121, compl 2, extd 30
Apr 7 20:47:14 blade2 kernel: [ 76.679082] be2net 0000:02:00.1: Error
in cmd completion - opcode 121, compl 2, extd 30
Из собственного BIOS контроллер видит таргеты и LUNы в них.
Резюмируя написанное выше, хочу попросить помощи в запуске этой железяки
в наших дистрибутивах. Заранее спасибо.
PS iopen-iscsi версии 2.0-872 собрал сам. Наш 2.0-871 не работает с
be2iscsi вообще. Дистрибутив - Server Light на p6, ядра из сизифа.
--
WBR, Alex moskalenko
Подробная информация о списке рассылки Hardware