Clustered OpenAM 13.5 Deployment: OpenAM Nodes going out of sync

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Clustered OpenAM 13.5 Deployment: OpenAM Nodes going out of sync

Manish Sharma
Hello,

We have a clustered (two node) OpenAM Deployment, connected to a single OpenDJ. There is an ILB in front of OpenAM nodes which does sticky routing based on "amlbcookie".

We are seeing an issue wherein when a realm is marked inactive, only the node which processed that request has the status of realm as inactive. If we access the same realm using the other OpenAM node, it returns the status as active and even returns a valid session when json/<realm>/authenticate API is executed on that second node. I have verified the status of realm is persisted correctly in OpenDJ. This behavior is not consistent and happens randomly.

Also, if we restart the second node, the issue goes away and now the second node also returns "Org is inactive". We have tried disabling the SM cache in OpenAM but the issue is still reproducible.

Has anybody else experienced this behavior or know a possible cause for this issue?

Thanks,
Manish

_______________________________________________
Visit the OpenAM forum at https://forgerock.org/forum/fr-projects/openam/
OpenAM mailing list
[hidden email]
https://lists.forgerock.org/mailman/listinfo/openam
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Clustered OpenAM 13.5 Deployment: OpenAM Nodes going out of sync

Vincent Koldenhof-2

Hi Manish,

Can you describe how you have setup the configuration store for OpenAM? Do you have a single configuration store used by both OpenAM nodes or do you have two configuration nodes which uses replication?

Kind regards,

Vincent


Op 10/05/2017 om 11:05 schreef Manish Sharma:
Hello,

We have a clustered (two node) OpenAM Deployment, connected to a single OpenDJ. There is an ILB in front of OpenAM nodes which does sticky routing based on "amlbcookie".

We are seeing an issue wherein when a realm is marked inactive, only the node which processed that request has the status of realm as inactive. If we access the same realm using the other OpenAM node, it returns the status as active and even returns a valid session when json/<realm>/authenticate API is executed on that second node. I have verified the status of realm is persisted correctly in OpenDJ. This behavior is not consistent and happens randomly.

Also, if we restart the second node, the issue goes away and now the second node also returns "Org is inactive". We have tried disabling the SM cache in OpenAM but the issue is still reproducible.

Has anybody else experienced this behavior or know a possible cause for this issue?

Thanks,
Manish


_______________________________________________
Visit the OpenAM forum at https://forgerock.org/forum/fr-projects/openam/
OpenAM mailing list
[hidden email]
https://lists.forgerock.org/mailman/listinfo/openam


_______________________________________________
Visit the OpenAM forum at https://forgerock.org/forum/fr-projects/openam/
OpenAM mailing list
[hidden email]
https://lists.forgerock.org/mailman/listinfo/openam
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Clustered OpenAM 13.5 Deployment: OpenAM Nodes going out of sync

Vincent Koldenhof-2
In reply to this post by Manish Sharma

Hi Manish,

Can you describe how you have setup the configuration store for OpenAM? Do you have a single configuration store used by both OpenAM nodes or do you have two configuration nodes which uses replication?

Kind regards,

Vincent


Op 10/05/2017 om 11:05 schreef Manish Sharma:
Hello,

We have a clustered (two node) OpenAM Deployment, connected to a single OpenDJ. There is an ILB in front of OpenAM nodes which does sticky routing based on "amlbcookie".

We are seeing an issue wherein when a realm is marked inactive, only the node which processed that request has the status of realm as inactive. If we access the same realm using the other OpenAM node, it returns the status as active and even returns a valid session when json/<realm>/authenticate API is executed on that second node. I have verified the status of realm is persisted correctly in OpenDJ. This behavior is not consistent and happens randomly.

Also, if we restart the second node, the issue goes away and now the second node also returns "Org is inactive". We have tried disabling the SM cache in OpenAM but the issue is still reproducible.

Has anybody else experienced this behavior or know a possible cause for this issue?

Thanks,
Manish


_______________________________________________
Visit the OpenAM forum at https://forgerock.org/forum/fr-projects/openam/
OpenAM mailing list
[hidden email]
https://lists.forgerock.org/mailman/listinfo/openam


_______________________________________________
Visit the OpenAM forum at https://forgerock.org/forum/fr-projects/openam/
OpenAM mailing list
[hidden email]
https://lists.forgerock.org/mailman/listinfo/openam
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Clustered OpenAM 13.5 Deployment: OpenAM Nodes going out of sync

Manish Sharma
In reply to this post by Vincent Koldenhof-2
Hi Vincent,

The configuration store is external. The same OpenDJ server and same base DN is used for both "configuration store" and "user store" (dc=example,dc=com). The first OpenAM server was installed in a standard way by giving the external OpenDJ information. The second node of OpenAM was installed by using add to existing deployment option (and giving the primary OpenAM's URL).


Thanks,
Manish

On 10 May 2017 at 14:42, Vincent Koldenhof <[hidden email]> wrote:

Hi Manish,

Can you describe how you have setup the configuration store for OpenAM? Do you have a single configuration store used by both OpenAM nodes or do you have two configuration nodes which uses replication?

Kind regards,

Vincent


Op 10/05/2017 om 11:05 schreef Manish Sharma:
Hello,

We have a clustered (two node) OpenAM Deployment, connected to a single OpenDJ. There is an ILB in front of OpenAM nodes which does sticky routing based on "amlbcookie".

We are seeing an issue wherein when a realm is marked inactive, only the node which processed that request has the status of realm as inactive. If we access the same realm using the other OpenAM node, it returns the status as active and even returns a valid session when json/<realm>/authenticate API is executed on that second node. I have verified the status of realm is persisted correctly in OpenDJ. This behavior is not consistent and happens randomly.

Also, if we restart the second node, the issue goes away and now the second node also returns "Org is inactive". We have tried disabling the SM cache in OpenAM but the issue is still reproducible.

Has anybody else experienced this behavior or know a possible cause for this issue?

Thanks,
Manish


_______________________________________________
Visit the OpenAM forum at https://forgerock.org/forum/fr-projects/openam/
OpenAM mailing list
[hidden email]
https://lists.forgerock.org/mailman/listinfo/openam


_______________________________________________
Visit the OpenAM forum at https://forgerock.org/forum/fr-projects/openam/
OpenAM mailing list
[hidden email]
https://lists.forgerock.org/mailman/listinfo/openam



_______________________________________________
Visit the OpenAM forum at https://forgerock.org/forum/fr-projects/openam/
OpenAM mailing list
[hidden email]
https://lists.forgerock.org/mailman/listinfo/openam
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Clustered OpenAM 13.5 Deployment: OpenAM Nodes going out of sync

Bernhard Thalmayr
check IdRepo debug log of the 2nd OpenAM instance when you change a
setting on the 1st OpenAM instance

If you don't see an persistent search result being handled by the
EventService then there is an issue with persistent search.

-Bernhard

Am 10/05/17 um 12:21 schrieb Manish Sharma:

> Hi Vincent,
>
> The configuration store is external. The same OpenDJ server and same
> base DN is used for both "configuration store" and "user store"
> (dc=example,dc=com). The first OpenAM server was installed in a standard
> way by giving the external OpenDJ information. The second node of OpenAM
> was installed by using add to existing deployment option (and giving the
> primary OpenAM's URL).
>
>
> Thanks,
> Manish
>
> On 10 May 2017 at 14:42, Vincent Koldenhof <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Hi Manish,
>
>     Can you describe how you have setup the configuration store for
>     OpenAM? Do you have a single configuration store used by both OpenAM
>     nodes or do you have two configuration nodes which uses replication?
>
>     Kind regards,
>
>     Vincent
>
>
>     Op 10/05/2017 om 11:05 schreef Manish Sharma:
>>     Hello,
>>
>>     We have a clustered (two node) OpenAM Deployment, connected to a
>>     single OpenDJ. There is an ILB in front of OpenAM nodes which does
>>     sticky routing based on "amlbcookie".
>>
>>     We are seeing an issue wherein when a realm is marked inactive,
>>     only the node which processed that request has the status of realm
>>     as inactive. If we access the same realm using the other OpenAM
>>     node, it returns the status as active and even returns a valid
>>     session when json/<realm>/authenticate API is executed on that
>>     second node. I have verified the status of realm is persisted
>>     correctly in OpenDJ. This behavior is not consistent and happens
>>     randomly.
>>
>>     Also, if we restart the second node, the issue goes away and now
>>     the second node also returns "Org is inactive". We have tried
>>     disabling the SM cache in OpenAM but the issue is still reproducible.
>>
>>     Has anybody else experienced this behavior or know a possible
>>     cause for this issue?
>>
>>     Thanks,
>>     Manish
>>
>>
>>     _______________________________________________
>>     Visit the OpenAM forum at https://forgerock.org/forum/fr-projects/openam/
>>     <https://forgerock.org/forum/fr-projects/openam/>
>>     OpenAM mailing list
>>     [hidden email] <mailto:[hidden email]>
>>     https://lists.forgerock.org/mailman/listinfo/openam
>>     <https://lists.forgerock.org/mailman/listinfo/openam>
>
>
>     _______________________________________________
>     Visit the OpenAM forum at
>     https://forgerock.org/forum/fr-projects/openam/
>     <https://forgerock.org/forum/fr-projects/openam/>
>     OpenAM mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.forgerock.org/mailman/listinfo/openam
>     <https://lists.forgerock.org/mailman/listinfo/openam>
>
>
>
>
> _______________________________________________
> Visit the OpenAM forum at https://forgerock.org/forum/fr-projects/openam/
> OpenAM mailing list
> [hidden email]
> https://lists.forgerock.org/mailman/listinfo/openam
>


--
Painstaking Minds
IT-Consulting Bernhard Thalmayr
Herxheimer Str. 5, 83620 Vagen (Munich area), Germany
Tel: +49 (0)8062 7769174
Mobile: +49 (0)176 55060699

[hidden email] - Solution Architect
http://www.xing.com/profile/Bernhard_Thalmayr
http://de.linkedin.com/in/bernhardthalmayr

This e-mail may contain confidential and/or privileged information.If
you are not the intended recipient (or have received this email in
error) please notify the sender immediately and delete this e-mail. Any
unauthorized copying, disclosure or distribution of the material in this
e-mail is strictly forbidden.
_______________________________________________
Visit the OpenAM forum at https://forgerock.org/forum/fr-projects/openam/
OpenAM mailing list
[hidden email]
https://lists.forgerock.org/mailman/listinfo/openam
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Clustered OpenAM 13.5 Deployment: OpenAM Nodes going out of sync

Andy Cory-2
We’ve also had issues with persistent searches, the resulting symptoms similar to yours. https://bugster.forgerock.org/jira/browse/OPENAM-10852 and https://bugster.forgerock.org/jira/browse/OPENAM-10800 were relevant in our case.

Andy

On 16/05/2017, 20:24, "[hidden email] on behalf of Bernhard Thalmayr" <[hidden email] on behalf of [hidden email]> wrote:

    check IdRepo debug log of the 2nd OpenAM instance when you change a
    setting on the 1st OpenAM instance

    If you don't see an persistent search result being handled by the
    EventService then there is an issue with persistent search.

    -Bernhard

    Am 10/05/17 um 12:21 schrieb Manish Sharma:
    > Hi Vincent,
    >
    > The configuration store is external. The same OpenDJ server and same
    > base DN is used for both "configuration store" and "user store"
    > (dc=example,dc=com). The first OpenAM server was installed in a standard
    > way by giving the external OpenDJ information. The second node of OpenAM
    > was installed by using add to existing deployment option (and giving the
    > primary OpenAM's URL).
    >
    >
    > Thanks,
    > Manish
    >
    > On 10 May 2017 at 14:42, Vincent Koldenhof <[hidden email]
    > <mailto:[hidden email]>> wrote:
    >
    >     Hi Manish,
    >
    >     Can you describe how you have setup the configuration store for
    >     OpenAM? Do you have a single configuration store used by both OpenAM
    >     nodes or do you have two configuration nodes which uses replication?
    >
    >     Kind regards,
    >
    >     Vincent
    >
    >
    >     Op 10/05/2017 om 11:05 schreef Manish Sharma:
    >>     Hello,
    >>
    >>     We have a clustered (two node) OpenAM Deployment, connected to a
    >>     single OpenDJ. There is an ILB in front of OpenAM nodes which does
    >>     sticky routing based on "amlbcookie".
    >>
    >>     We are seeing an issue wherein when a realm is marked inactive,
    >>     only the node which processed that request has the status of realm
    >>     as inactive. If we access the same realm using the other OpenAM
    >>     node, it returns the status as active and even returns a valid
    >>     session when json/<realm>/authenticate API is executed on that
    >>     second node. I have verified the status of realm is persisted
    >>     correctly in OpenDJ. This behavior is not consistent and happens
    >>     randomly.
    >>
    >>     Also, if we restart the second node, the issue goes away and now
    >>     the second node also returns "Org is inactive". We have tried
    >>     disabling the SM cache in OpenAM but the issue is still reproducible.
    >>
    >>     Has anybody else experienced this behavior or know a possible
    >>     cause for this issue?
    >>
    >>     Thanks,
    >>     Manish
    >>
    >>
    >>     _______________________________________________
    >>     Visit the OpenAM forum at https://forgerock.org/forum/fr-projects/openam/
    >>     <https://forgerock.org/forum/fr-projects/openam/>
    >>     OpenAM mailing list
    >>     [hidden email] <mailto:[hidden email]>
    >>     https://lists.forgerock.org/mailman/listinfo/openam
    >>     <https://lists.forgerock.org/mailman/listinfo/openam>
    >
    >
    >     _______________________________________________
    >     Visit the OpenAM forum at
    >     https://forgerock.org/forum/fr-projects/openam/
    >     <https://forgerock.org/forum/fr-projects/openam/>
    >     OpenAM mailing list
    >     [hidden email] <mailto:[hidden email]>
    >     https://lists.forgerock.org/mailman/listinfo/openam
    >     <https://lists.forgerock.org/mailman/listinfo/openam>
    >
    >
    >
    >
    > _______________________________________________
    > Visit the OpenAM forum at https://forgerock.org/forum/fr-projects/openam/
    > OpenAM mailing list
    > [hidden email]
    > https://lists.forgerock.org/mailman/listinfo/openam
    >


    --
    Painstaking Minds
    IT-Consulting Bernhard Thalmayr
    Herxheimer Str. 5, 83620 Vagen (Munich area), Germany
    Tel: +49 (0)8062 7769174
    Mobile: +49 (0)176 55060699

    [hidden email] - Solution Architect
    http://www.xing.com/profile/Bernhard_Thalmayr
    http://de.linkedin.com/in/bernhardthalmayr

    This e-mail may contain confidential and/or privileged information.If
    you are not the intended recipient (or have received this email in
    error) please notify the sender immediately and delete this e-mail. Any
    unauthorized copying, disclosure or distribution of the material in this
    e-mail is strictly forbidden.
    _______________________________________________
    Visit the OpenAM forum at https://forgerock.org/forum/fr-projects/openam/
    OpenAM mailing list
    [hidden email]
    https://lists.forgerock.org/mailman/listinfo/openam






This email has been scanned for all viruses.

Please consider the environment before printing this email.

The content of this email and any attachment is private and may be privileged. If you are not the intended recipient, any use, disclosure, copying or forwarding of this email and/or its attachments is unauthorised. If you have received this email in error please notify the sender by email and delete this message and any attachments immediately. Nothing in this email shall bind the Company or any of its subsidiaries or businesses in any contract or obligation, unless we have specifically agreed to be bound.

KCOM Group PLC is a public limited company incorporated in England and Wales, company number 02150618 and whose registered office is at 37 Carr Lane, Hull, HU1 3RE.

_______________________________________________
Visit the OpenAM forum at https://forgerock.org/forum/fr-projects/openam/
OpenAM mailing list
[hidden email]
https://lists.forgerock.org/mailman/listinfo/openam
Loading...