I would suggest going with `FROM golang-alpine` instead, instantly gives you a much smaller image without needing to do much yourself. It's still going to be a bit bigger than a manually crafted image but it's probably small enough yet you can still use anything packages for Alpine.
If you want to go super-tiny you can also just `FROM scratch` instead which for a "hello world" Go app would clock in at about 1.2MB.