One node crashing on startup with " a bytes-like object is required, not 'str'"

Node Operators troubleshooting
1

I have a docker compose stack with like 5 different nodes. this morning I was greeted with ONE node which was failed. when I try to start the stack I get this error and the whole stack fails.

xxx@sdocker:~$ sudo docker-compose up
storj14004 is up-to-date
storj14003 is up-to-date
Starting storj14007 ...
storj14002 is up-to-date
storj14005 is up-to-date

ERROR: for storj14007  a bytes-like object is required, not 'str'

ERROR: for storj14007  a bytes-like object is required, not 'str'
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/docker/api/client.py", line 261, in _raise_for_status
    response.raise_for_status()
  File "/usr/lib/python3/dist-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.22/containers/2c843c59a2d942606a9fe76ad5c9960f193333fce71c469b42035d05d16245df/start

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/compose/service.py", line 625, in start_container
    container.start()
  File "/usr/lib/python3/dist-packages/compose/container.py", line 241, in start
    return self.client.start(self.id, **options)
  File "/usr/lib/python3/dist-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/docker/api/container.py", line 1095, in start
    self._raise_for_status(res)
  File "/usr/lib/python3/dist-packages/docker/api/client.py", line 263, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/lib/python3/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 500 Server Error: Internal Server Error ("b"error while creating mount source path '/srv/storj4': mkdir /srv/storj4: file exists"")

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/docker-compose", line 11, in <module>
    load_entry_point('docker-compose==1.25.0', 'console_scripts', 'docker-compose')()
  File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 72, in main
    command()
  File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 128, in perform_command
    handler(command, command_options)
  File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 1107, in up
    to_attach = up(False)
  File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 1088, in up
    return self.project.up(
  File "/usr/lib/python3/dist-packages/compose/project.py", line 565, in up
    results, errors = parallel.parallel_execute(
  File "/usr/lib/python3/dist-packages/compose/parallel.py", line 112, in parallel_execute
    raise error_to_reraise
  File "/usr/lib/python3/dist-packages/compose/parallel.py", line 210, in producer
    result = func(obj)
  File "/usr/lib/python3/dist-packages/compose/project.py", line 548, in do
    return service.execute_convergence_plan(
  File "/usr/lib/python3/dist-packages/compose/service.py", line 567, in execute_convergence_plan
    return self._execute_convergence_start(
  File "/usr/lib/python3/dist-packages/compose/service.py", line 506, in _execute_convergence_start
    _, errors = parallel_execute(
  File "/usr/lib/python3/dist-packages/compose/parallel.py", line 112, in parallel_execute
    raise error_to_reraise
  File "/usr/lib/python3/dist-packages/compose/parallel.py", line 210, in producer
    result = func(obj)
  File "/usr/lib/python3/dist-packages/compose/service.py", line 508, in <lambda>
    lambda c: self.start_container_if_stopped(c, attach_logs=not detached, quiet=True),
  File "/usr/lib/python3/dist-packages/compose/service.py", line 620, in start_container_if_stopped
    return self.start_container(container)
  File "/usr/lib/python3/dist-packages/compose/service.py", line 627, in start_container
    if "driver failed programming external connectivity" in ex.explanation:
TypeError: a bytes-like object is required, not 'str'

I dunno what to make of it but wanted to copy paste it quickly.

2

From what you’ve posted, it seems like the issue might be related to port usage. A similar error has been discussed in this GitHub issue, where it turned out that a port being used by the failing container was still occupied on the host machine.

I’d suggest checking the ports defined for storj14007 in your docker-compose.yml file and ensuring they are free on your host. You can use a command like netstat -tuln or lsof -i to check if the ports are in use. If they are, stopping the conflicting process or assigning a different port in your stack might resolve the problem.

Let us know how it goes, and keep us posted if you need further help! You’ve got this! :blush:

1 Like
3

Accordingly the issue posted by @edo

this error is a result of a missing destination port.
The example:

    ports:
      - "28972:28967/tcp"
      - "28972:28967/udp"
      - "127.0.0.1:14005:14002"

But to be sure it would be better, if you could post your docker-compose.yaml here between two new lines with three backticks like this:

```
    ports:
      - "28972:28967/tcp"
      - "28972:28967/udp"
      - "14005:14002"
```

then it would be pretty formatted in the post. You may mask any personal information.

4

Thank you gentlemen, false alarm apparently.

Netstat and lsof didn’t show anything.

Looks like the underlying hard disk for the node was having a problem.

Powering down, jiggling the power cable, and rebooting the server fixed it.

Here was the docker port setup for that node btw, nothing fancy.

    ports:
      - "28971:28967/udp"
      - "28971:28967/tcp"
      - "14007:14002"
2 Likes