Most don’t think too much about income taxes in retirement. They just assume that they’ll pay less since it’s possible (likely?) that their income will be less. For many, that’s a reasonable assumption (at least as long as tax rates remain relatively low). Others believe that their Social Security retirement benefits aren’t taxable, further reducing their overall tax bill.

Well, as it turns outs, some of us won’t have to pay taxes on Social Security (SS) benefits, but many of us will. If you start receiving benefits and have a pension or annuity, withdraw from savings, or work part-time, you probably will. And as we will see, how much tax you pay depends on a variety of factors.

Out of sight, out of mind

I have to confess that I don’t think too much about income taxes. I didn’t think about them too much before I retired either except to consider whether a Roth conversion for some of our savings in a traditional (taxable) IRA would make sense for us (it didn’t).

Yet, I sometimes think I should have given it more consideration before I retired and come up with ways to generate more tax-free income in retirement. Still, other than Roth conversions, there aren’t many other options that don’t involve insurance products like permanent life and non-qualified annuities (i.e., held outside of a retirement savings account).

My wife and I are receiving SS benefits, and those combined with withdrawals from our IRA are the money we live on. I don’t have taxes withheld from our SS payments, but the Feds take out our Medicare Part B premiums. However, every time I withdraw money from my taxable IRA account, I set aside at least 15 to 20 percent for state and local taxes. Fidelity withholds those sums for me and reports them on a 1099 at the end of the year.

I know I have to pay taxes, and I should do so cheerfully and with gratitude (Rom. 13:1–7), but it stings a little because, for every $1.00 that I withdraw, I can only spend 80 to 85 cents of it.

Why do we have to pay taxes?

SS has been running on fumes for a while now. And back in the 1980s, Congress passed legislation to try to shore it up. One of the more controversial additions to the Amendments of a 1983 bill was taxing Social Security benefits, which the government officially implemented in 1984. It was intended to only impact upper-income senior households, and its purpose was to help raise additional revenue and avoid reducing retiree benefits.

It has been pretty unpopular ever since.

If most of our income came from Social Security, we probably wouldn’t owe any tax. That’s good news and bad, of course. It would be nice not to have to pay taxes but perhaps not so good that we’d have little more than Social Security to live on. I think it’s better to have a little more than just Social Security, even if I have to pay taxes on it. Of course, the more income other than SS we have, the more taxes we will have to pay.

We have to pay taxes because, 1) Most of our SS benefits are taxable due to income from savings that pushes our total income over certain limits, and 2) The money I withdraw from savings is taxable whether we owe taxes on our SS benefits or not.

Almost any form of income in addition to SS benefits except Roth IRA distributions would push us over the limits, but these are the most common ones:

• Large IRA account balances
• Large taxable (non-retirement) account balances
• Large investments in municipal bonds (non-taxable interest)
• Pensions
• Annuities
• Employment
• Inflation (more on that shortly)

Some of our SS benefits are taxable because our “provisional income” exceeds a “base amount” of $32,000 a year (for married filing jointly; it’s $25,000 a year for singles). If we were below that amount, none of our SS benefits would be taxed.

What’s ”provisional income”?

You may be wondering, “What’s provisional income’”? The IRS defines it as the amount of income used to determine how much of your SS benefit is taxable. It’s the sum of half of your SS benefits plus Adjusted Gross Income (AGI) from your federal tax return plus non-taxable interest.

For example, let’s say that your SS benefits are $45,000 per year, and you have an AGI of $35,000 due to withdrawals from a traditional IRA. Your total income (before taxes) would be $80,000 ($45,000 + $35,000 = $80,000); but your provisional income would be $57,000:

(SS benefits ÷ 2) + AGI + Non-taxable interest; and $22,500 + $35,000 + 0 = $57,500.

Provisional income of $57,500 is $25,500 over the IRS’ $32,000 limit, suggesting that most of your SS benefits will be taxable. (We’ll learn by how much in the next section, but here’s a nice little calculator if you want to play with the numbers for yourself.)

It’s important to note that Adjusted Gross Income (AGI) used to determine your provisional income is the sum of all your annual income; it’s taxable income BEFORE any deductions (standard or itemized) and includes things like wages, capital gains, dividends, pensions, interest, IRA distributions and just about any other taxable income source.

Interestingly, AGI doesn’t include tax-free interest, but provisional income does. But most people won’t have tax-free income unless they own or invest in things like municipal bonds.

My traditional IRA distributions are included in our AGI when I take money out of my IRA (which most will eventually have to do due to Required Minimum Distributions), making some of our SS benefits taxable. It makes no difference whether I only withdraw interest and dividends or sell investment assets and spend the proceeds—it’s all treated as taxable income.

I have a relatively small balance in a Roth IRA. Distributions from a Roth IRA are not included in provisional income. But since I’m just letting it grow (i.e., not taking distributions), it’s not of any help tax-wise right now.

So, in my case (and for many other retirees), it’s tough to keep your SS benefits from being taxable. Furthermore, if your SS benefits become taxable, it may increase your marginal tax rate (more on that in a bit).

How much tax?

Retirees may prefer not to pay any taxes on their SS benefits. If you’d rather not pay taxes on annual SS benefits of $45,000/year, because of the IRS limit, you could only withdraw $9,500 per year from taxable savings or receive it as other taxable income (such as part-time work):

Annual limit ($32,000) minus 50% of your SS benefits ($22,500) = $9,500.

We can deduce that those with smaller SS benefits can have more income from the sources already mentioned without their SS benefits being taxed. While the rule favors those with lower benefits, especially with less income from other sources, the opposite is also true: Those with larger benefits and more additional income will pay more tax.

The IRS rules state that for combined incomes between $25,000 to $34,000 (for singles) and $32,000 to $44,000 (for married filing jointly), $0.50 of every dollar above the base amount of $32,000 up to $44,000 will be taxable, equating to 50 percent.

Note that only the portion of your provisional income that exceeds the base amount is taxable. Just because your provisional income exceeds the $32,000 hurdle doesn’t mean all of your benefits will be taxable at 50%. Only the amount over $32,000 is taxable at that rate, up to the next hurdle, which is $44,000.

And once your combined income reaches the top of the range ($32,000 for a single filer or $44,000 for married filing jointly), $0.85 of every dollar of your SS benefits will be taxable (equating to 85%). But it’s important to understand that this doesn’t mean you’re going to pay 50% or 85% of your benefits in taxes—it’s the amount of your benefits that will be included as part of your taxable income on your tax return.

Confused? Returning to our earlier example may help.

With “provisional income” of $57,500 and based on a married filing jointly status, the first $32,000 of your SS benefits won’t be taxed. However, $0.50 of every dollar of combined income between $32,000 and $44,000 ($12,000) is taxable.

That leaves an additional $13,500 ($25,500 – $12,000 = $13,500) that’s over the $44,000 limit. Therefore, $0.85 of each dollar (or 85% of that amount—$11,475 of your SS benefits), will also be taxable.

In this example, your total taxable SS benefits will therefore be 50% of $12,000 (or, $6,000) plus 85% of $13,500 (or, $11,475), which is equal to $17,475, or approx. 39% of taxable benefits.

If we add that to AGI of $35,000 (from IRA withdrawals), you’d have a total AGI of $46,475. After taking the standard deduction for married filing jointly ($25,100), you’ll have a taxable income of $18,675 and a tax liability of $1,868. That’s an effective tax rate of only 2.3% on a total income of $80,000!

Marginal rates?

How much this all matters to you (i.e., paying taxes on some part of your SS benefits) is a personal matter. However, it might be a greater concern for retirees whose SS benefits suddenly become taxable, effectively pushing them into a higher tax bracket (i.e., a higher marginal tax rate).

It’s wise for all retirees to watch their marginal tax rates, which is the amount of tax you pay on your last dollar of income. In the above example, with a taxable income of $21,375 based on a total retirement income of $80,000, your effective tax rate is 2.7%. However, you’re in the 12% marginal bracket, which is the rate at which your next taxable dollar of income will be taxed:

• For income between $0 – $19,900, you pay 10% ($1,990)
• For income between $19,901 – $81,050, you pay $1,990 + 12% of the amount over $19,900 ($21,375 – $19,900 = $1,475)

The next marginal bracket is 22%, but it doesn’t apply in this case since it doesn’t kick in until your taxable income exceeds $81,050:

• For income between $81,051 – $172,750, you pay $9,328 + 22% of the amount over $81,050

This illustrates that, for the vast majority of retirees with taxable incomes below $81,050, the taxation of SS benefits will basically be a non-issue in terms of total tax liability. Moreover, if you are married and filing jointly and your provisional income is below $32,000, none of your SS benefits will be taxable.

However, once you have enough AGI that your SS benefits become taxable, that additional taxable income can push your marginal tax bracket significantly higher. Why? Well, remember how the IRS determines how much of your SS benefit is taxable.

If you are a married couple and receive $40,000 in combined Social Security benefits:

• None of your benefits are taxable if your other income is less than $12,000.
• For every dollar between $12,000 and $24,000, an additional 50 cents becomes taxable.
• For every dollar over $24,000, an additional 85 cents becomes taxable, up to a total other income of $56,941, which makes the maximum $34,000 taxable.

Why is this a concern? When Social Security benefits become taxable, the additional taxable income can push your marginal tax bracket significantly higher.

Another factor that can cause your SS benefits to become taxable and increase your marginal tax rate is taking Required Minimum Distributions (RMDs) from your taxable IRA accounts when you reach age 72.

Inflation is another important consideration when analyzing the taxability of SS benefits. SS benefits generally rise to keep up (somewhat) with inflation; however, the provisional income-based amounts ($25,000/$32,000 for single and $32,000/$44,000 for married filing jointly) do not.

Inflation is running high right now, and if it continues at 3% a year for the next ten years and Congress doesn’t change the law, that lower limit will be $18,600 in today’s dollars. In 20 years, it will be less than $14,000. Your retirement could last even longer than that, and that limit will become easier and easier to surpass.

I admit this is complicated and confusing, so use the calculator I alluded to earlier to get familiar with the possibilities and, as always, consult with a tax professional if you need to develop a plan to minimize your taxes.

What to do?

If your goal is to avoid paying any tax on any of your SS benefits, but you have other sources of retirement income, you’ll have to keep an eye on things every year.

Benefits could be taxable one year when you work part-time or withdraw money from your IRA to buy a car but not the next year when you don’t. Or, they may not be taxable if you don’t start withdrawing from your IRA until age 72 when you must take RMDs.

I’m not a tax expert, and I haven’t found a way to avoid paying taxes on our SS benefits. But if you are trying to prevent it, there may be some things you can do. They mainly involve tracking your additional income carefully to ensure you don’t exceed the maximum allowed before your SS benefits become taxable. Otherwise, render Ceasar his due.