Neighbor wants me arrested for taking my package

Been here for 3 years, my house looks similar to my neighbors. We never really talk to each, just few times we’ve said hello casually. We’ve never had any dispute or issues. Anyway, I order from Amazon a lot, 90% of the time they deliver to my address, but 10% of the time they deliver it to my neighbors address, we both have big address number in front, but I’m guessing delivery drivers are being careless/ rush sometimes?

Most of the time my neighbor just brings it over and drop it off at my door. Twice I went over and grabbed it by his porch. (expensive items) he never said anything, once I tried ringing just to let him know but he never answered.

Yesterday they delivered to the wrong address again (1st time in about 5 months now), I waited about 24h, he didn’t drop it off, so Today I went over to his porch and grabbed my package.

About 2 hours later, he dropped off a note in my mailbox, he didn’t knock or anything, note said “dont you ever fucking go on my property to get misaddressed delivery or I’ll have you arrested, I have cameras”.

Obviously I hate going to his property, and it’s a hassle for both of us. I’ve noticed on my ring camera when he drops my package in the past he’s always annoyed. But it’s not my fault they keep delivering to the wrong address, package clearly says my name and my address, I not sure Amazon can do anything as they always have different drivers.

What would you do in this situation? I’m going to bring him the note and try to have a chat, I get his point, but I personally don’t mind if my neighbor goes to my porch to get his mail if delivered wrong, but he never orders from Amazon so it’s one-sided situation.