Probably, it could poke its insides you should take it to the vet.
Given that it is a solid object that who degrade and emit toxins and that it is a small piece, your puppy most likely will be fine and within a day or two it should come out in a BM.
However, do watch you puppy closely. If he suddenly doesn't want to eat his food, eats his food and then vomits shortly after, starts acting tired or getting "lazy", or does "dry heaves"...tries to vomit or gag but nothing comes up....then he needs to go to the vet ASAP. These are symptoms of a foreign body obstruction. As for it the object could cut open the digestive track...you would be surprised how strong that system is, so it is unlikely. The systems of this are the same as foreign body obstruction but also add in noticable painfulness and potential vomiting of blood. However, unless an object was really big, this is very unlikely to happen. I actually saw a yarn sewing needle make it all the way through a cats digestive system and came out in the kitties litterbox with a short piece of yarn still threaded.
So, as long as your puppy remains a happy go lucky puppy for the next couple day, then all is good. Even if you don't see it in the poo, after a max of 4 days if no symptoms happen, then they wont and it passed and you just didn't see it.
Good luck and don't worry too much. This actually was a relatively safe thing compared to the many things our pets think are edible.