ns3 simulation end with an "illegal instruction"

[alienpj]

when following tutorial simai-simulation usage, running command:
python3 ./astra-sim-alibabacloud/inputs/topo/gen_Topo_Template.py -topo Spectrum-X -g 8 -psn 1
generating topo file successfully:
asw_switch_num: 8
psw_switch_num: 1
Creating Topology of totally 1 segment(s), totally 1 pod(s).
Spectrum-X_8g_8gps_400Gbps_H100

however while running simai-ns3 simulation with command:
AS_SEND_LAT=3 AS_NVLS_ENABLE=1 ./bin/SimAI_simulator -t 16 -w ./example/microAllReduce.txt -n ./Spectrum-X_8g_8gps_400Gbps_H100 -c astra-sim-alibabacloud/inputs/config/SimAI.conf

a Segmentation fault (core dumped) occur :
ps_H100 -c astra-sim-alibabacloud/inputs/config/SimAI.conf
maxRtt=150 maxBdp=54000
Segmentation fault (core dumped)

I try to recompile the ns3 environment with a sudo:
sudo ./scripts/build.sh -c ns3
the problem still happen.

then try running the simai-ns3 simulation with a sudo:
sudo AS_SEND_LAT=3 AS_NVLS_ENABLE=1 ./bin/SimAI_simulator -t 16 -w ./example/microAllReduce.txt -n ./Spectrum-X_8g_8gps_400Gbps_H100 -c astra-sim-alibabacloud/inputs/config/SimAI.conf
then it comes another problem that the simulation end abnormally:
maxRtt=150 maxBdp=54000
Running Simulation.
The final active chunks per dimension 1 after allocating to queues is: 1
ring of node 0, id: 0 dimension: local total nodes in ring: 9 index in ring: 0 offset: 1total nodes in ring: 9
ring of node 0, id: 0 dimension: local total nodes in ring: 9 index in ring: 0 offset: 1total nodes in ring: 9
ring of node 0, id: 0 dimension: local total nodes in ring: 9 index in ring: 0 offset: 1total nodes in ring: 9
ring of node 0, id: 0 dimension: local total nodes in ring: 9 index in ring: 0 offset: 1total nodes in ring: 9
total nodes: 9
Success in opening workload file
model_parallel_NPU_group is 8
checkpoints layers are:
layers initiating fwd_in_bckwd are:
ring of node 0, id: 0 dimension: local total nodes in ring: 8 index in ring: 0 offset: 1total nodes in ring: 8
ring of node 0, id: 0 dimension: local total nodes in ring: 1 index in ring: 0 offset: 8total nodes in ring: 1
ring of node 0, id: 0 dimension: local total nodes in ring: 8 index in ring: 0 offset: 1total nodes in ring: 8
ring of node 0, id: 0 dimension: local total nodes in ring: 1 index in ring: 0 offset: 8total nodes in ring: 1
ring of node 0, id: 0 dimension: local total nodes in ring: 8 index in ring: 0 offset: 1total nodes in ring: 8
ring of node 0, id: 0 dimension: local total nodes in ring: 1 index in ring: 0 offset: 8total nodes in ring: 1
ring of node 0, id: 0 dimension: local total nodes in ring: 8 index in ring: 0 offset: 1total nodes in ring: 8
ring of node 0, id: 0 dimension: local total nodes in ring: 1 index in ring: 0 offset: 8total nodes in ring: 1
pp_commize:0
id: embedding_layer , depen: -1 , wg_comp_time: 1
id: embedding_layer , depen: -1 , wg_comp_time: 1
type: HYBRID_TRANSFORMER_FWD_IN_BCKWD ,num passes: 1 ,lines: 2 compute scale: 1 ,comm scale: 1
stat path: ./ncclFlowModel_ ,total rows: 1 ,stat row: 0
CSV path and filename: ./ncclFlowModel_detailed_9.csv
CSV path and filename: ./ncclFlowModel_EndToEnd.csv
simulator run
chunk size is: 16777216 , size is: 16777216 , layer_num is: 0 , node: 0
info: all-reduce forward pass collective issued for layer: embedding_layer, involved dimensions: 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,
Illegal instruction

the output file "ncclFlowModel_detailed.csv" and "ncclFlowModel_EndToEnd.csv" are empty (0 byte).

is this some environment problem associate with docker version?

[Yan824]

Hi, the core dump might be caused by the location permission of the output files in SimAI.conf (/etc/astra-sim/simulation/). Please first use gdb ./bin/SimAI_simulator core.yourpid to confirm the cause.

[alienpj]

Hi, the core dump might be caused by the location permission of the output files in SimAI.conf (/etc/astra-sim/simulation/). Please first use gdb ./bin/SimAI_simulator core.yourpid to confirm the cause.

Thanks very much for your advice, I've checked my /astra-sim repository, it's in /home/eddy-huang/SimAI/astra-sim-alibabacloud/astra-sim ,however there is not a folder named "simulation" in it , there are only there folders in it: network_frontend, system, workload.

is it generated after running the command:./scripts/build.sh -c ns3? does it means that i have some other compile problem?

[Yan824]

Hi, the core dump might be caused by the location permission of the output files in SimAI.conf (/etc/astra-sim/simulation/). Please first use gdb ./bin/SimAI_simulator core.yourpid to confirm the cause.

Thanks very much for your advice, I've checked my /astra-sim repository, it's in /home/eddy-huang/SimAI/astra-sim-alibabacloud/astra-sim ,however there is not a folder named "simulation" in it , there are only there folders in it: network_frontend, system, workload.

is it generated after running the command:./scripts/build.sh -c ns3? does it means that i have some other compile problem?

These files are generated during your simulation, not compile time. And files start with /etc/ are system files, you can use ls /etc/astra-sim/ to see whether these file exist. And if you don't have root permission, you can try to modified the path in SimAI.conf.

[alienpj]

Hi, the core dump might be caused by the location permission of the output files in SimAI.conf (/etc/astra-sim/simulation/). Please first use gdb ./bin/SimAI_simulator core.yourpid to confirm the cause.

Thanks very much for your advice, I've checked my /astra-sim repository, it's in /home/eddy-huang/SimAI/astra-sim-alibabacloud/astra-sim ,however there is not a folder named "simulation" in it , there are only there folders in it: network_frontend, system, workload.
is it generated after running the command:./scripts/build.sh -c ns3? does it means that i have some other compile problem?

These files are generated during your simulation, not compile time. And files start with /etc/ are system files, you can use ls /etc/astra-sim/ to see whether these file exist. And if you don't have root permission, you can try to modified the path in SimAI.conf.

again, thank you for the advice . After switching to the root user , I recompile the simai-simulation ns3 , and then run the demo , the simulation still interrupted abnormally with the output:
root@eddy-huang-mgpusim:~/SimAI# AS_SEND_LAT=12 AS_NVLS_ENABLE=1 ./bin/SimAI_simulator -t 8 -w ./example/microAllReduce.txt -n ./Rail_Opti_SingleToR_32g_8gps_400Gbps_H100 -c ./astra-sim-alibabacloud/inputs/config/SimAI.conf
maxRtt=4720 maxBdp=236000
Running Simulation.
The final active chunks per dimension 1 after allocating to queues is: 1
ring of node 0, id: 0 dimension: local total nodes in ring: 36 index in ring: 0 offset: 1total nodes in ring: 36
ring of node 0, id: 0 dimension: local total nodes in ring: 36 index in ring: 0 offset: 1total nodes in ring: 36
ring of node 0, id: 0 dimension: local total nodes in ring: 36 index in ring: 0 offset: 1total nodes in ring: 36
ring of node 0, id: 0 dimension: local total nodes in ring: 36 index in ring: 0 offset: 1total nodes in ring: 36
total nodes: 36
Success in opening workload file
model_parallel_NPU_group is 8
checkpoints layers are:
layers initiating fwd_in_bckwd are:
ring of node 0, id: 0 dimension: local total nodes in ring: 8 index in ring: 0 offset: 1total nodes in ring: 8
ring of node 0, id: 0 dimension: local total nodes in ring: 4 index in ring: 0 offset: 8total nodes in ring: 4
ring of node 0, id: 0 dimension: local total nodes in ring: 8 index in ring: 0 offset: 1total nodes in ring: 8
ring of node 0, id: 0 dimension: local total nodes in ring: 4 index in ring: 0 offset: 8total nodes in ring: 4
ring of node 0, id: 0 dimension: local total nodes in ring: 8 index in ring: 0 offset: 1total nodes in ring: 8
ring of node 0, id: 0 dimension: local total nodes in ring: 4 index in ring: 0 offset: 8total nodes in ring: 4
ring of node 0, id: 0 dimension: local total nodes in ring: 8 index in ring: 0 offset: 1total nodes in ring: 8
ring of node 0, id: 0 dimension: local total nodes in ring: 4 index in ring: 0 offset: 8total nodes in ring: 4
pp_commize:0
id: embedding_layer , depen: -1 , wg_comp_time: 1
id: embedding_layer , depen: -1 , wg_comp_time: 1
type: HYBRID_TRANSFORMER_FWD_IN_BCKWD ,num passes: 1 ,lines: 2 compute scale: 1 ,comm scale: 1
stat path: ./ncclFlowModel_ ,total rows: 1 ,stat row: 0
CSV path and filename: ./ncclFlowModel_detailed_36.csv
CSV path and filename: ./ncclFlowModel_EndToEnd.csv
simulator run
chunk size is: 16777216 , size is: 16777216 , layer_num is: 0 , node: 0
info: all-reduce forward pass collective issued for layer: embedding_layer, involved dimensions: 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,
Illegal instruction (core dumped)

I do check the system file :
root@eddy-huang-mgpusim:~/SimAI# ls /etc/astra-sim/
config inputs results SimAI.log simulation topo

except the SimAI.log file have something written in it , other folders are all empty.

[alienpj]

Hi, the core dump might be caused by the location permission of the output files in SimAI.conf (/etc/astra-sim/simulation/). Please first use gdb ./bin/SimAI_simulator core.yourpid to confirm the cause.

try to use gbd to locate the problem:

root@eddy-huang-mgpusim:~/SimAI# gdb ./bin/SimAI_simulator /tmp/coredumps/core.SimAI_simulator.30504.eddy-huang-mgpusim.1746847002

comes the output:

Core was generated by `./bin/SimAI_simulator -t 16 -w ./example/microAllReduce.txt -n ./Spectrum-X_8g_'.
Program terminated with signal SIGILL, Illegal instruction.
#0 0x0000768e7c7a1c61 in ns3::RdmaHw::SendPacketComplete (
this=0x5b0a64795730, p=..., ch=...)
at /root/SimAI/astra-sim-alibabacloud/extern/network_backend/ns3-interface/simulation/src/point-to-point/model/rdma-hw.cc:406
--Type for more, q to quit, c to continue without paging--display/i $pc
406 }
[Current thread is 1 (Thread 0x768e7ccec740 (LWP 30504))]
(gdb) bt
#0 0x0000768e7c7a1c61 in ns3::RdmaHw::SendPacketComplete (
this=0x5b0a64795730, p=..., ch=...)
at /root/SimAI/astra-sim-alibabacloud/extern/network_backend/ns3-interface/simulation/src/point-to-point/model/rdma-hw.cc:406
#1 0x0000768e7c7b8fe1 in ns3::MemPtrCallbackImpl<ns3::RdmaHw*, int (ns3::RdmaHw::)(ns3::Ptrns3::Packet, ns3::CustomHeader&), int, ns3::Ptrns3::Packet, ns3::CustomHeader&, ns3::empty, ns3::empty, ns3::empty, ns3::empty, ns3::empty, ns3::empty, ns3::empty>::operator() (this=0x5b0a64795cc0, a1=..., a2=...)
at /root/SimAI/astra-sim-alibabacloud/extern/network_backend/ns3-interface/simulation/build/include/ns3/callback.h:636
#2 0x0000768e7c77dc6f in ns3::Callback<int, ns3::Ptrns3::Packet, ns3::CustomHeader&, ns3::empty, ns3::empty, ns3::empty, ns3::empty, ns3::empty, ns3::empty, ns3::empty>::operator() (this=0x5b0a64544858, a1=..., a2=...)
at /root/SimAI/astra-sim-alibabacloud/extern/network_backend/ns3-interface/simulation/build/include/ns3/callback.h:1420
#3 0x0000768e7c7791d7 in ns3::QbbNetDevice::SendCallback (
this=0x5b0a645445d0, packet=...)
at /root/SimAI/astra-sim-alibabacloud/extern/network_backend/ns3-interface/simulation/src/point-to-point/model/qbb-net-device.cc:589
#4 0x0000768e7c780ce7 in ns3::MakeEvent<void (ns3::QbbNetDevice::)(ns3::Ptrns3::Packet), ns3::QbbNetDevice*, ns3::Ptrns3::Packet >(void (ns3::QbbNetDevice::)(ns3::Ptrns3::Packet), ns3::QbbNetDevice, ns3::Ptrns3::Packet)::EventMemberImpl1::Notify() (this=0x5b0a6481ac90)

the gbd output illustrates that it trigger a "SIGILL" error when calling the function ns3::RdmaHw::SendPacketComplete , seems the program attempts to execute an instruction that is not supported by the CPU?

[WWeiOne]

I ran into the same issue using Ubuntu 24.04.
This problem did not occur when I was using Ubuntu 20.04.