As you might imagine, our judicial system is not perfect and errors are made. Thus, sometimes people can be wrongfully convicted. It’s that plain and simple.
Even though you aren't using it, you can (and often will) be charged.
What that means is if you are near someone with marijuana (or anything illegal for that matter), you need to remove yourself from the situation. If someone is doing something illegal, similarly you need to remove yourself from the situation.
Call an attorney to see if you have a defense.
If you've been charged for possessing marijuana that wasn't yours, you should talk to an attorney immediately, be fully open with him or her, and let them know particular facts that you would think would be helpful in your defense. From that, defense counsel will be able to advise you of whether you have a solid defense at trial. To convict you, the government must prove you guilty beyond a reasonable doubt.
There are issues that need to be addressed before making a decision about your case.
Defense counsel will also need to be made aware of the facts that lead you to encounter law enforcement to determine if there are any constitutional issues that were violated. What is more, counsel will thoroughly review other defenses, such as whether proper testing was conducted of the substance to prove it was actually marijuana.
Related Question: What does the Commonwealth attorney in Virginia have to prove in marijuana cases?