Hi guys, I get many requests from 79.127.220.97 to my node on port 28967 which cause network spike. Anyone knows what this IP is used for? Should I block it?
Thank you,
I guess this IP is the hosted gateway. It if obviously a STORJ IP address. Please open the IP in your browser and you will land at the STORJ website.
I would not block it if you care about your node.
3 Likes
Well I got a flood of UDP packets from that IP which can be considered as a DOS attack. Why? Anyone from Storj can give me some hints? Thank you.
Can you give any extra details about the nature or the size of the flood?
In fact I get massive of TCP SYN packets from those IPs even when the node is turned off, which is strange
135.181.183.90
79.127.220.97
5.161.71.1
58.186.149.69
…
about 10k packets in 1 minute (from all those IPs), but it much slower now since I stopped the node. Shouldn’t all traffic be stopped when the node is turned off?
There shouldn’t be any traffic from the satellite, once it counts your node as offline (which could be hours). That could be traffic from the gateway, though, if there is a spike in downloads of some popular segment which has pieces on your node.
That seems like too much, though. Any chance you can get a packet capture?
1 Like
At least two of those IPs don’t belong to us, I’m pretty sure. They could be normal Storj clients. All of them spiked in number of packets sent?
Yeah it stopped, some hours after the node is offline. I don’t know the exact time when it stopped because I was sleeping.
Nah the network was so laggy that I had to stop playing to check what causes the lag… So I run tcpdump for 3 or 4 times, 1 min duration each, output on screen so no logs. It looked like TCP SYN flood. Then I turned the node off and went to sleep.
There’s different between packets from 79.127.220.97 (which I think is from Storj) and the rest of them. That’s all I can remember.
Anyway it stopped now. I’ll start the node again. If it happens again I’ll update this topic. Thank you.
@Alexey man I need your help. “max-concurrent-requests” this param still can be used?
Lol. It’s back but at a very slow rate. Ping check? and I still haven’t restarted the node. The node has been offline for 5 hours. IP 37.59.37.96, very similar pattern to previous packets. I have found the IP in a thread on Storj forum ( VPN issue with port forwarding? - Node Operators / troubleshooting - Storj Community Forum (official)). So it seems to be a satellite.
18:39:11.713341 IP 37.59.37.96.56914 > x.x.x.x.28967: Flags [S], seq 2274817642, win 29200, options [mss 1460,sackOK,TS val 936963520 ecr 0,nop,wscale 7], length 0
0x0000: 4500 003c 10f7 4000 3706 79a1 253b 2560 E..<..@.7.y.%;%`
0x0010: c108 ad80 de52 7127 8796 f66a 0000 0000 .....Rq'...j....
0x0020: a002 7210 29b7 0000 0204 05b4 0402 080a ..r.)...........
0x0030: 37d8 edc0 0000 0000 0103 0307 7...........
I think I’ll use “max-concurrent-requests” to limit the request per minute. The line is not dedicated for Storj using.
Ok. Let us know here if you see another SYN flood and can capture the packets.
You can do this, but note that max-concurrent-requests will only limit how many connections the node will accept at a time, not how many connections can be requested. So it won’t limit the rate of SYN packets received; it will just make the node reject (with FIN) any additional connections after the specified number of connections is reached. It may harm your node reputation if you reject connections from an auditor.
Edited to add: 37.59.37.96 is the IP of a proxy in France (for a node which sounds like it’s in Russia), and not the satellite. But there shouldn’t be any need for that node to connect to yours, so this must be uplink traffic through that same proxy.
3 Likes
Yeah max-concurrent-requests doesn’t help. Just checking another node and I got like 25k TCP packets on port 28967 in 1 minute. Is that normal?
Command used: tcpdump -i ens3 -n 'dst port 28967'
Sample
1:23:18.658713 IP 58.186.149.69.34246 > x.x.x.x.28967: Flags [.], seq 12740:14014, ack 1, win 502, options [nop,nop,TS val 4215467581 ecr 2211867757], length 1274
21:23:18.658737 IP 58.186.149.69.34246 > x.x.x.x.28967: Flags [P.], seq 14014:15074, ack 1, win 502, options [nop,nop,TS val 4215467581 ecr 2211867757], length 1060
21:23:18.673150 IP 157.90.209.184.57822 > x.x.x.x.28967: Flags [P.], seq 95550:98098, ack 1, win 502, options [nop,nop,TS val 1477852179 ecr 1967467592], length 2548
21:23:18.698903 IP 157.90.209.184.57822 > x.x.x.x.28967: Flags [P.], seq 98098:100646, ack 1, win 502, options [nop,nop,TS val 1477852198 ecr 1967467611], length 2548
21:23:18.705166 IP 157.90.209.184.57822 > x.x.x.x.28967: Flags [.], seq 100646:101920, ack 1, win 502, options [nop,nop,TS val 1477852198 ecr 1967467611], length 1274
21:23:18.728363 IP 157.90.209.184.57822 > x.x.x.x.28967: Flags [P.], seq 101920:104468, ack 1, win 502, options [nop,nop,TS val 1477852231 ecr 1967467643], length 2548
21:23:18.737282 IP 189.191.152.246.54348 > x.x.x.x.28967: Flags [.], seq 20384:22932, ack 1, win 502, options [nop,nop,TS val 483722562 ecr 3565920262], length 2548
21:23:18.737285 IP 189.191.152.246.54348 > x.x.x.x.28967: Flags [.], seq 22932:24206, ack 1, win 502, options [nop,nop,TS val 483722562 ecr 3565920262], length 1274
21:23:18.747943 IP 157.90.209.184.57822 > x.x.x.x.28967: Flags [P.], seq 104468:107016, ack 1, win 502, options [nop,nop,TS val 1477852256 ecr 1967467669], length 2548
21:23:18.753268 IP 67.159.32.114.47920 > x.x.x.x.28967: Flags [.], seq 53508:54782, ack 1, win 280, options [nop,nop,TS val 1055486972 ecr 1940352306], length 1274
21:23:18.753512 IP 67.159.32.114.47920 > x.x.x.x.28967: Flags [P.], seq 54782:57330, ack 1, win 280, options [nop,nop,TS val 1055486973 ecr 1940352306], length 2548
21:23:18.753911 IP 67.159.32.114.47920 > x.x.x.x.28967: Flags [P.], seq 57330:59878, ack 1, win 280, options [nop,nop,TS val 1055486973 ecr 1940352307], length 2548
21:23:18.753957 IP 67.159.32.114.47920 > x.x.x.x.28967: Flags [P.], seq 59878:62426, ack 1, win 280, options [nop,nop,TS val 1055486973 ecr 1940352307], length 2548
21:23:18.754323 IP 67.159.32.114.47920 > x.x.x.x.28967: Flags [P.], seq 62426:64974, ack 1, win 280, options [nop,nop,TS val 1055486974 ecr 1940352307], length 2548
21:23:18.754407 IP 67.159.32.114.47920 > x.x.x.x.28967: Flags [P.], seq 64974:65633, ack 1, win 280, options [nop,nop,TS val 1055486974 ecr 1940352307], length 659
21:23:18.763109 IP 79.127.220.97.51386 > x.x.x.x.28967: Flags [P.], seq 0:24, ack 5972, win 16, options [nop,nop,TS val 2405267394 ecr 1102887928], length 24
21:23:18.764382 IP 79.127.220.97.51386 > x.x.x.x.28967: Flags [F.], seq 24, ack 5972, win 16, options [nop,nop,TS val 2405267395 ecr 1102887928], length 0
I tried to set a counter with iptables and got same same result. This node is in the UK.
25k packets per minute seems pretty reasonable. If the packet sizes you show are representative of the packets in general, then they have an average size of about 1700 B. Multiplying that by 25k and the number of minutes in a day, it would be about 60GB/day of ingress, or a sustained average rate of 700kB/s. Does that sound right?
3 Likes