Fernandina Beach is on the border with Georgia. It's like you described but kind of dead and with absolutely no industry. Due to the lack of industry they have a slightly touristy main street but its not busy just cheesey. Sebastian is a great place for surfing and fishing! The best for both in Florida. It's south of Melbourne. Not real touristy or busy. St. Augustine beaches are nice but St. A is completely devoted to tourism. It's probably not what your looking for. When I was a kid we used to go up to Cape Hatteras, in August. I love that place but the water is real cold right now. Panama City is not a bad choice, its probably going to be gearing up but at a low time for tourists, in May.
Have a nice time!