Cannot remove 2nd log file on AlwaysOn database

Hi all,

I have a database, member of an availability group. This database has 2 log file, I want to remove the unsed secondary log file, I try to run this command to empty the second lofg file:

USE [TEST-AG]
GO
DBCC SHRINKFILE (N'TEST-AG_log_2' , EMPTYFILE)
GO

the command completes successfully, the I run the command to remove the file:

USE [TEST-AG]
GO
ALTER DATABASE [TEST-AG]  REMOVE FILE [TEST-AG_log_2]
GO

But this command fails with the following message:

Error 5042: The file 'TEST-AG_log_2' cannot be removed because it is not empty.

If I remove the database from availability group the command to remove the 2nd file works, so I can't remove a secondary log file on a database member of an alwayson availability grup?